<html>
<head><meta charset="utf-8"><title>crates.io index http mirror · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html">crates.io index http mirror</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="218775795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218775795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218775795">(Dec 04 2020 at 00:53)</a>:</h4>
<p><span class="user-mention" data-user-id="120054">@Jon Gjengset</span> has raised in <a class="stream-topic" data-stream-id="246057" href="/#narrow/stream/246057-t-cargo/topic/http.20registry.20RFC">#t-cargo &gt; http registry RFC</a> that it'd be good for a soon-to-be-merged RFC to have a real endpoint for folks to test against, before we migrate <a href="http://crates.io">crates.io</a> itself to that new API</p>



<a name="218775817"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218775817" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218775817">(Dec 04 2020 at 00:53)</a>:</h4>
<blockquote>
<p>I do think it would be valuable to have a real endpoint that people can try out after my PR lands so that people can test, but that's relatively low priority. To make it low effort, that should probably just be a CloudFront in front of <a href="http://raw.githubusercontent.com">raw.githubusercontent.com</a> with a relatively short cache expiry time, so that we don't have to implement any purging or S3 bucket updating.</p>
</blockquote>



<a name="218775863"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218775863" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218775863">(Dec 04 2020 at 00:54)</a>:</h4>
<p>I am a bit concerned about making requests direct to github</p>



<a name="218775904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218775904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218775904">(Dec 04 2020 at 00:55)</a>:</h4>
<p>I think what I'd do is setup a tiny ec2 instance with nginx serving a checkout of the index and a cronjob refreshing it every 5 minutes or so</p>



<a name="218776182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776182">(Dec 04 2020 at 00:59)</a>:</h4>
<p>Hehe, that would work too I suppose. My guess would be that proxying to GitHub would be fine _if_ we did explicit cache eviction as opposed to a timeout, but with a TTL I agree it's probably better to proxy to something under <a href="http://crates.io">crates.io</a> control.</p>



<a name="218776272"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776272" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776272">(Dec 04 2020 at 01:00)</a>:</h4>
<p>I don't know that we have fine-grained access to caching like that for cloudfront</p>



<a name="218776278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776278">(Dec 04 2020 at 01:00)</a>:</h4>
<p>I do think that medium-term you'd want <a href="http://crates.io">crates.io</a> to, on publish, go poke both the checkout server to do a pull and the CloudFront cache to evict the index file of the published crate. But that's probably not needed for initial experimentation. It just means people will sometimes get very slightly stale views of the index, which is fine.</p>



<a name="218776300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776300">(Dec 04 2020 at 01:01)</a>:</h4>
<p>I think Pietro mentioned that there is indeed a cache shootdown API for cloudfront</p>



<a name="218776378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776378">(Dec 04 2020 at 01:02)</a>:</h4>
<p>Cache invalidations exist on cloudfront, but they're somewhat expensive IIRC</p>



<a name="218776419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776419">(Dec 04 2020 at 01:03)</a>:</h4>
<p>$0.005/invalidation path (which can be <code>*</code>)</p>



<a name="218776546"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218776546" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218776546">(Dec 04 2020 at 01:05)</a>:</h4>
<p>Huh, interesting. That's a bummer. Well, even more reason to stick to simple cache expiry for the initial test then. I'll go poke at some things "behind the scenes" at AWS too and see what I can find out there.</p>



<a name="218777164"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218777164" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218777164">(Dec 04 2020 at 01:16)</a>:</h4>
<p>Are you able to set short TTLs for the CloudFront caching at the moment? Or is it something silly like "minimum of 1hr TTL"?</p>



<a name="218777334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218777334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218777334">(Dec 04 2020 at 01:19)</a>:</h4>
<p>From <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#ExpirationDownloadDist">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html#ExpirationDownloadDist</a> it _seems_ like we can set TTL basically as low as we want, so at least that's good.</p>



<a name="218778449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218778449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218778449">(Dec 04 2020 at 01:40)</a>:</h4>
<p><a href="https://aws.amazon.com/about-aws/whats-new/2010/08/31/cloudfront-adds-invalidation-feature/">$0.005 charge for invalidating each file</a> * <a href="https://github.com/rust-lang/crates.io-index/graphs/commit-activity">2644 commits to the index per week</a> = 13.22 US$ per week.</p>



<a name="218778920"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218778920" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218778920">(Dec 04 2020 at 01:48)</a>:</h4>
<p>Yeah, it's not too bad -- only a couple extra hundred per year -- but I'd rather not add it just because :)</p>



<a name="218778937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218778937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218778937">(Dec 04 2020 at 01:48)</a>:</h4>
<p>I think short TTLs would work -- I've long wanted to do some investigation/reading on how much CloudFront caching in front of S3 buys you</p>



<a name="218778962"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218778962" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218778962">(Dec 04 2020 at 01:49)</a>:</h4>
<p>I think it's probably hard to measure in the US since our S3 buckets all live in us-west-2 I think</p>



<a name="218784427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218784427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218784427">(Dec 04 2020 at 02:33)</a>:</h4>
<p>The main thing it would buy you is lower tail latency for users (they won't miss in cache unless the index file has _just_ been updated), and the ability for users to use a new crate version just _slightly_ faster (on average half the TTL). Is that worth it? Unclear...</p>



<a name="218817020"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218817020" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218817020">(Dec 04 2020 at 10:46)</a>:</h4>
<p>I think for <a href="http://crates.io">crates.io</a> the best way would be to actually just have the <a href="http://crates.io">crates.io</a> backend update s3</p>



<a name="218817106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218817106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218817106">(Dec 04 2020 at 10:47)</a>:</h4>
<p>the requirement for that is to implement crate deletions on the application, instead of the current approach of "pietro manually pushes a commit to the index"</p>



<a name="218817865"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218817865" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218817865">(Dec 04 2020 at 10:55)</a>:</h4>
<p>in general <span class="user-mention" data-user-id="120054">@Jon Gjengset</span> I think the question of <a href="http://crates.io">crates.io</a> implementing it needs to be done to the <a href="http://crates.io">crates.io</a> team</p>



<a name="218817875"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218817875" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218817875">(Dec 04 2020 at 10:55)</a>:</h4>
<p>I know jtgeibel has opinions on this too</p>



<a name="218834446"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218834446" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218834446">(Dec 04 2020 at 13:49)</a>:</h4>
<p>yeah so to be clear I think that's absolutely the right way</p>



<a name="218834481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218834481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218834481">(Dec 04 2020 at 13:49)</a>:</h4>
<p>but I was thinking that a temporary solution might be worthwhile if <a href="http://crates.io">crates.io</a> doesn't have time</p>



<a name="218837677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218837677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218837677">(Dec 04 2020 at 14:18)</a>:</h4>
<p>I don't think we should expect <a href="http://crates.io">crates.io</a> to instantly implement that RFC</p>



<a name="218837787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218837787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218837787">(Dec 04 2020 at 14:19)</a>:</h4>
<p>both because we need to make sure <a href="http://crates.io">crates.io</a> can scale</p>



<a name="218837814"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218837814" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218837814">(Dec 04 2020 at 14:19)</a>:</h4>
<p>and also because <a href="http://crates.io">crates.io</a> currently has <em>very little</em> human bandwidth at the moment to design, implement and review</p>



<a name="218838266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218838266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218838266">(Dec 04 2020 at 14:23)</a>:</h4>
<p>Right, hence trying to think how we can get some more testing while requiring as little from <a href="http://crates.io">crates.io</a> as possible.</p>



<a name="218844998"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218844998" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218844998">(Dec 04 2020 at 15:14)</a>:</h4>
<p>We have a bit of a chicken/egg thing developing:</p>
<ul>
<li>it is hard to get widespread testing without a production-ish server somewhere.</li>
<li>it is (legitimately) not worth <a href="http://crates.io">crates.io</a> teams time to set up a server for an unproven protocol.</li>
</ul>
<p>So we need to find some way to bootstrap, a server that is good enough to test on but is not much work for <a href="http://crates.io">crates.io</a>.</p>



<a name="218847472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218847472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218847472">(Dec 04 2020 at 15:31)</a>:</h4>
<p>yeah, and I think my solution is both low effort and well, is a static file server so scales well</p>



<a name="218873040"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218873040" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218873040">(Dec 04 2020 at 18:29)</a>:</h4>
<p>My sense from talking to the CloudFront team is that 3k invalidations per week is such a low number that it's unlikely to get any kind of "special treatment" :p</p>



<a name="218873809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218873809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218873809">(Dec 04 2020 at 18:35)</a>:</h4>
<p>not unexpected</p>



<a name="218873814"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218873814" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218873814">(Dec 04 2020 at 18:35)</a>:</h4>
<p>I'm not sold we need it</p>



<a name="218873835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218873835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218873835">(Dec 04 2020 at 18:35)</a>:</h4>
<p>I do think figuring out what an invalidation <em>means</em> might be nice</p>



<a name="218873874"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218873874" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218873874">(Dec 04 2020 at 18:35)</a>:</h4>
<p>if that just kills our cache, then maybe we can set a TTL of 0 or something and get roughly the same effects</p>



<a name="218873886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218873886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218873886">(Dec 04 2020 at 18:36)</a>:</h4>
<p>(presuming we invalidate everything)</p>



<a name="218874459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218874459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218874459">(Dec 04 2020 at 18:40)</a>:</h4>
<p>The other way to look at this is that if specific invalidations means most things can have much longer TTLs, that might already make up for itself in terms of cost. Missing in cache is more expensive than hitting in cache after all.</p>



<a name="218874633"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218874633" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218874633">(Dec 04 2020 at 18:41)</a>:</h4>
<p>It would be interesting to try to compare the two</p>



<a name="218874758"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218874758" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218874758">(Dec 04 2020 at 18:42)</a>:</h4>
<p>I imagine it's going to end up being that missing is actually not that expensive cost wise (unclear on time though)</p>



<a name="218874763"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218874763" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218874763">(Dec 04 2020 at 18:42)</a>:</h4>
<p>Another point the CloudFront team brought up to me was that the number of invalidations scales with the number of _active_ crates, not the number of crates total, which means that it'll likely have a lower rate than overall <a href="http://crates.io">crates.io</a> size.</p>



<a name="218874893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218874893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218874893">(Dec 04 2020 at 18:43)</a>:</h4>
<p>Yeah, could be, it's hard to say. But I imagine that the difference between a TTL=5m and no invalidations and TTL=3h + explicit invalidations makes quite the difference in terms of number of requests to the origin server. Not sure if CF also charges more for origin requests?</p>



<a name="218874956"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218874956" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218874956">(Dec 04 2020 at 18:43)</a>:</h4>
<p>As far as I know we only pay the S3 GET cost</p>



<a name="218875060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875060">(Dec 04 2020 at 18:44)</a>:</h4>
<blockquote>
<p>not the number of crates total, which means that it'll likely have a lower rate than overall <a href="http://crates.io">crates.io</a> size</p>
</blockquote>
<p>The data is available of commits per day. It is no longer in github ui, but I can pull it together if you want it.</p>



<a name="218875128"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875128" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875128">(Dec 04 2020 at 18:45)</a>:</h4>
<p>One thing that would also be super useful is to spin up a server in say australia or something far away and measure latency on invalidated requests</p>



<a name="218875159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875159">(Dec 04 2020 at 18:45)</a>:</h4>
<p>i.e. run your benchmarks against cold cloudfront or ttl cloudfront</p>



<a name="218875204"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875204" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875204">(Dec 04 2020 at 18:45)</a>:</h4>
<p>Yeah, that would be interesting too.</p>



<a name="218875288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875288">(Dec 04 2020 at 18:46)</a>:</h4>
<p>Overall I feel like the experience of hitting cache often will be nicer for users, but I don't have direct data to that effect.</p>



<a name="218875360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875360">(Dec 04 2020 at 18:46)</a>:</h4>
<p>Yep, that would be my theory but would be nice to know how much that benefit is - if it's, say, 2x then I'm willing to invest in it</p>



<a name="218875440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875440">(Dec 04 2020 at 18:47)</a>:</h4>
<p>But if we win very little then invalidating might be just simpler model (since it gives us strong consistency presumably)</p>



<a name="218875654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875654">(Dec 04 2020 at 18:49)</a>:</h4>
<p>Not sure I follow: "win" in what regard? cost? I think the upsides of low TTL are reduced complexity and (maybe) slightly lower cost. And the upsides of invalidations are stronger consistency (no need to wait for TTL + retry), and likely lower tail latency.</p>



<a name="218875881"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875881" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875881">(Dec 04 2020 at 18:51)</a>:</h4>
<p>No, time (ms)</p>



<a name="218875977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218875977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218875977">(Dec 04 2020 at 18:52)</a>:</h4>
<p>Hm, I am surprised by the lower tail latency</p>



<a name="218876043"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876043" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876043">(Dec 04 2020 at 18:53)</a>:</h4>
<p>I guess it depends on if we invalidate in a targeted manner</p>



<a name="218876066"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876066" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876066">(Dec 04 2020 at 18:53)</a>:</h4>
<p>I know for static.r-l.o we basically just invalidate the whole bucket</p>



<a name="218876078"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876078" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876078">(Dec 04 2020 at 18:53)</a>:</h4>
<p>/dist/* iirc</p>



<a name="218876124"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876124" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876124">(Dec 04 2020 at 18:53)</a>:</h4>
<p>Which - who knows - might actually be biting us a bunch</p>



<a name="218876543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876543">(Dec 04 2020 at 18:57)</a>:</h4>
<p>Ah, no, I think for invalidations we'd specifically want targeted invalidations. Blanket invalidations would mean you probably do _worse_ than TTL depending on update frequency, at higher cost.</p>



<a name="218876632"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876632" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876632">(Dec 04 2020 at 18:57)</a>:</h4>
<p>Lower tail latency because you'd hit CloudFront not the origin server more often, and the whole point of CloudFront is to cache to speed up access :p</p>



<a name="218876691"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876691" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876691">(Dec 04 2020 at 18:58)</a>:</h4>
<p>I guess it also depends on the relative cost of S3 access vs CloudFront access</p>



<a name="218876744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218876744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218876744">(Dec 04 2020 at 18:58)</a>:</h4>
<p>(assuming the index would also be served through an S3 bucket)</p>



<a name="218878073"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218878073" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218878073">(Dec 04 2020 at 19:09)</a>:</h4>
<p>Another interesting data point in favor of TTL: CloudFront supports conditional requests (<code>If-Modified-Since</code>) to the origin server, so even with a low TTL it won't actually pull much _data_ from the backend server for things that expire but do not change. Would still inflict an incremental latency cost though.</p>



<a name="218878153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218878153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218878153">(Dec 04 2020 at 19:10)</a>:</h4>
<p>Unless CloudFront checks with the origin server "in the background" rather than on a request, which I'm not sure about.</p>



<a name="218879684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218879684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218879684">(Dec 04 2020 at 19:23)</a>:</h4>
<p>Overall, I think low TTL caching is probably the way to go for now unless we explicitly find that it's not good enough. Eventually, if we add a signed snapshot of the index with content-addressed index files like Pietro has been thinking about, that would in turn enable us to give all the index files a super long TTL (since they'd be immutable), and we'd _only_ need to think about the snapshot file.</p>



<a name="218882769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218882769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218882769">(Dec 04 2020 at 19:50)</a>:</h4>
<blockquote>
<p>not the number of crates total, which means that it'll likely have a lower rate than overall <a href="http://crates.io">crates.io</a> size</p>
</blockquote>
<p>I ran some numbers. The commits per day is doubling every 1.75 years (R^2 = 0.94). That is slower than the number of crates witch is doubling every 1.5 years.</p>



<a name="218902578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218902578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218902578">(Dec 04 2020 at 23:05)</a>:</h4>
<p>my thinking on this FWIW is that if we don't do invalidations, the 5m cache time is good. Otherwise with invalidations which should have "near infinite" cache times because it will be valid until we invalidate it. And yeah the number of changes to the index is O(publishes), not O(size of registry)</p>



<a name="218963929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218963929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218963929">(Dec 06 2020 at 02:00)</a>:</h4>
<p>Hello from the CloudFront team!<br>
I discussed this a bit with Jon yesterday. Both the low-ttl variant and invalidations should be feasible strategies.</p>
<p>One thing that they all can't address is however having a mixed view of the index. <br>
E.g. let's say a user builds something that uses hyper. The index for that is fetched - but it's not in cache - so we get a fresh version from the origin which tells that now a version requiring tokio 0.3 is available. Then the resolver tries to fetch tokio - but the index file for this is in cache and only lists version 0.2 ==&gt; Build fails, unless there are any retries (e.g. using if-modified-since headers).<br>
It's unlikely to happen - but just like all flaky and racy things it will happen, and make peoples builds fail. We all know those things are annoying, and it would be good to avoid it.</p>
<p>We discussed that the easiest way to get around this is probably to have an "index of index". E.g. the a first HTTP request gives you the current index snapshot number (let's say <code>4123</code>). Then all follow-up requests go towards ( <code>/4123/path_to_crate</code>) instead of directly <code>path_to_crate</code>.<br>
 That makes sure that the client always has a consistent view of the index, and neither a TTL nor invalidations would be required for the index snapshots.</p>
<p>There are obviously some downsides to it:</p>
<ul>
<li>1 more RTT --&gt; I'm not convinced it matters. The index of index files should be highly cached since everyone needs it. So in most cases we are only talking about 1 round trip to an edge-location which is very close to the user. And I guess there are multiple layers of follow-up lookups, so that a +1 doesn't add that much (but Jon likely has numbers on that)</li>
<li>Who cleans up the old indexes? If the origin is S3, one could likely just use lifecycle rules for this which automatically delete files older than X (where X should here be bigger than the time <a href="http://crates.io">crates.io</a> usually publishes new snapshots).</li>
<li>Creating a snapshot requires copying O(N) packages, and N is growing exponentially. That is unfortunately true. But depending on the index size it might not matter at the moment. An optimization can be: On the authorative origin, don't copy files into the snapshot folders, but maintain the as symlink collections. E.g. <code>/4123/to/ki/tokio</code> points to <code>/tokio/up_to_0.3</code>. That way building a new snapshot mainly requires adding symlinks to the same revision. However one would require a custom origin for this. Whether it's worthwhile to go that route is likely a tradeoff between engineering complexity and cost. The preferred solution might change over time. Given the current solution is "make a copy of the full index on each checkout", and this would change it to "make a copy of the full index on each build" - it seems like the copy will be tolerable for some time.</li>
</ul>
<p>In case a custom origin is chosen instead of S3, and there are concerns about high load on this origin, one could look into using the "CloudFront Origin Shield feature", which adds a central cache layer between the edge locations and the authorative origin - which means the actual origin won't be queried by N edge locations anymore. But my feeling is that it might not be required for this use-case, since serving small static files should be rather efficient.</p>



<a name="218964853"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218964853" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218964853">(Dec 06 2020 at 02:31)</a>:</h4>
<p>After finding the comment thread at <a href="https://github.com/rust-lang/rfcs/pull/2789/files#r536821549">https://github.com/rust-lang/rfcs/pull/2789/files#r536821549</a> about this topic: Yes, retrying with a non-cacheable request can definitely also work! However in this case more of the complexity lives on the client side.</p>



<a name="218966973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218966973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218966973">(Dec 06 2020 at 03:42)</a>:</h4>
<p>We can use the snapshot number for <a href="https://github.com/rust-lang/cargo/pull/8890#issuecomment-738181228">RegistryData::current_version</a>, so that is an advantage.  Do you have a link for me to read up on what makes things non-cacheable? One thought is to add <code>?snapshot-number=4123</code> to the end of all file requests. That will force the cache to fetch each file once per publish, without the host needing to know how to serve old versions of the files. But I don't know the rules well enough to know if that makes sense.</p>



<a name="218967645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/218967645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#218967645">(Dec 06 2020 at 04:04)</a>:</h4>
<p>Whether query strings are part of the cache-key (lead to separately cached versions of files) or not is now configurable via cache-policies:<br>
<a href="https://aws.amazon.com/de/blogs/networking-and-content-delivery/amazon-cloudfront-announces-cache-and-origin-request-policies/">https://aws.amazon.com/de/blogs/networking-and-content-delivery/amazon-cloudfront-announces-cache-and-origin-request-policies/</a><br>
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/QueryStringParameters.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/QueryStringParameters.html</a><br>
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html</a></p>
<p>So you can indeed add this kind of snapshot number and make sure you get a unique file. But that would require the origin to support the same. <br>
Or if it doesn't, it would fall back to supporting only one revision and lose the immutable property (which might be good enough for alternate registries if they can support immutability through other means).</p>



<a name="219123567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219123567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219123567">(Dec 07 2020 at 19:06)</a>:</h4>
<p>So, one concern with a query parameter is that some malicious user queries for a bunch of snapshot versions that that not yet happened yet to make the server cache a 404 (or worse yet, a stale view for that version). It can probably be worked around, but something worth keeping in mind.</p>



<a name="219124599"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219124599" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219124599">(Dec 07 2020 at 19:14)</a>:</h4>
<p>Isn't that an issue with any hash addressed system?</p>



<a name="219124935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219124935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219124935">(Dec 07 2020 at 19:16)</a>:</h4>
<p>No, it shouldn't be. If the storage is truly content-addressed, then trying to fetch a given file before it exists should result in a 404, and that 404 should be invalidated when the file does exist.</p>



<a name="219125177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219125177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219125177">(Dec 07 2020 at 19:18)</a>:</h4>
<p>What prevents the cash from serving the old 404 after the new content is on the origin?</p>



<a name="219125506"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219125506" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219125506">(Dec 07 2020 at 19:21)</a>:</h4>
<p>It's true that the 404 caching case is the same -- in both cases an invalidation has to happen for the hash when it occurs. The challenge is around serving someone a _stale_ version of a supposedly content-addressed file. If I request <code>/x/y?s=42</code> and the server just gives me whatever the current version of <code>/x/y</code> is (which may be <code>&lt;42</code> or even <code>&gt;42</code>), then things get awkward, because what I get back doesn't  match what I asked for</p>



<a name="219126399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219126399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219126399">(Dec 07 2020 at 19:28)</a>:</h4>
<p>True, It does not solve the security reason to have <code>hash addressing</code>, but it does solve the cash busting reason.</p>



<a name="219137116"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219137116" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219137116">(Dec 07 2020 at 20:56)</a>:</h4>
<p>You can control the lifetime of cached error responses separately from good responses. There shouldn't be a big issue if TTL for 4xx/5xx is e.g. &lt; 1min</p>



<a name="219137199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20index%20http%20mirror/near/219137199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20index.20http.20mirror.html#219137199">(Dec 07 2020 at 20:57)</a>:</h4>
<p>Default is 10s: <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages-expiration.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages-expiration.html</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>